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Key findings 


e Children have increased their overall knowledge of vocabulary by 50% when taking 
part in the Narrative Journey programme (increasing from 8.2 points on average 
before taking part in the programme to 12.3 points following the programme). 

e Children were better able to identify synonyms and word pairs for Narrative Journey 
words after taking part in the programme. 

e Analysis of children’s writing data suggests that children incorporated more 
Narrative Journey words but fewer synonyms in their writing after taking part in the 
programme. This might suggest that children have learned more of the words and 
want to use them in their writing. 

e In addition to increased vocabulary, after taking part in the programme, more 
children felt that pictures help them to remember new words and that if they know 
a lot of words, they can improve their writing. 

e Children who scored fewer marks in the pre-test increased their knowledge of 
vocabulary more than their higher-scoring peers. Their score saw a significantly 
greater increase in the word pairing task but not in the synonym task, suggesting 
that the word pairings might have been particularly beneficial to the lower-scoring 
children. 
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Introduction 

Mrs Wordsmith is a vocabulary programme that combines data science and creativity to 
show children the words they need for school and life. Hilariously illustrated by the Emmy 
award-winning artists behind Madagascar, this is a research-driven vocabulary product 
aiming to make learning vocabulary manageable and engaging. 


Developed with leading literacy experts, The Narrative Journey illustrates storytelling words 
for 6-13 years, with the aim to boost their reading and writing age. The words are grouped 
into six themes: Character, Weather, Taste and Smell, Action, Emotion and Settings. Each 
set includes a Word of the Day workbook and display stand, Word Pair cards and an 
illustrated dictionary. The classroom set also includes a teacher’s guide, which provides 
additional data and teaching ideas for each word. 


This pilot evaluation explored the benefit of the Narrative Journey on children’s vocabulary 
learning and their attitudes towards it within the classroom. The purpose of this evaluation 
was also to pilot the vocabulary test based on the Mrs Wordsmith product and the 
attitudinal questions so that they can be used to conduct a larger-scale evaluation of the 
product in the future. 


Methodology 

The National Literacy Trust was commissioned in April 2018 to conduct a pilot evaluation of 
the Narrative Journey used in four Oasis Academy schools at in Bristol over the summer 
term of 2018. 


Instrument 

The vocabulary test was created in collaboration with the Mrs Wordsmith schools team and 
the National Literacy Trust. The aim was to create a variety of tasks that measure children’s 
knowledge of vocabulary included in the Narrative Journey before and after taking part in 
the classroom-based vocabulary programme. The test was based solely on the character 
words in the pack to make learning all 48 of them over one term more manageable. 


Three vocabulary tasks were created: 


1. Image task: children were asked to match words to corresponding Mrs Wordsmith 
images from four multiple choice options. Eight words were included in the image 
task, chosen based on Mrs Wordsmith team’s expertise and experience to ensure 
that the task provides a reasonable challenge without being too difficult. 

2. Synonym task: children were asked to match a word to a word with the same 
meaning. Again, children worked with eight words and chose answers from four 
multiple choice options. 

3. Word pairing task: children were asked to match a total of eight words to words that 
often go with them in a sentence from a range of four multiple choice options. 
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In addition to the vocabulary test, the instrument included attitudinal statements about 
vocabulary learning. These were asked at the beginning of the test and measured on a five- 
point scale ranging from “strongly disagree” to “strongly agree”. The statements included in 
the instrument were: 


Learning new words is fun 

If | know a lot of words, then it can improve my writing 
| find it hard to learn new words 

Knowing a lot of words can make me a better reader 


Oo 0 0 0 O 


Pictures help me remember new words better 


The test was administered by teachers using a Google form. An initial coding error with the 
image task led to a loss of 90% of the data for this task in the pre-test. Although the error 
was fixed as soon as it was noticed, the significant loss at pre-test meant that this aspect of 
the instrument could not be assessed in this evaluation. 


Children completed a writing task before and after the programme, which was then 
returned to Mrs Wordsmith team for analysis. Pupils were given several prompts made up 
of words from the Narrative Journey, and chose from a variety of topics. Statistical methods 
were then used to assess whether children had used more words from the Narrative 
Journey character pack, their synonyms or collocations after taking part in the programme. 


Sample 

181 children filled the pre-test and 157 carried out the post-test. 122 responses from three 
schools matched in the pre- and post-test based on children’s initials and date of birth. 
Demographic information, such as gender and free school meal uptake, was not collected as 
part of the pilot and data from the writing task before and after the programme were only 
available for 54 children. 


Analysis 

The vocabulary test items were coded as “correct” and “incorrect” in both the pre- and 
post-test. The correct answers were awarded one point and incorrect answers no points. A 
maximum vocabulary score of 16 was available as there were a total of 16 items in the 
synonym and the word association tasks. This score was used to analyse the difference in 
children’s vocabulary knowledge before and after taking part in the activities. 


As well as looking at the overall vocabulary score, each of the two tasks, the synonym and 
word association tasks were also analysed separately. As both had eight words, the 
maximum score was eight points in each task. The analysis also explored the percentage of 
children who got each item correct in pre- and post-test. 


The attitudinal questions scale was coded from 1 (strongly disagree) to 5 (strongly agree) 
and the mean difference was explored between pre- and post-test for all the items 
individually. 


All findings presented in this report are statistically significant at the conventional 0.05 level. 
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Findings 

The data show that children have improved their knowledge of vocabulary over the course 
of the summer term. Before taking part in the programme, children scored an average of 8.2 
points out of 16 in the vocabulary test. After taking part, they scored an average of 12.3 
points; a mean difference of 4.1 points. This suggests that after the end of the programme, 
children had increased their vocabulary by 50%. 


Synonym task 
Children scored statistically significantly higher in the synonym task after taking part in the 


programme. Their average score increased from 4.3 in the pre-test to 6.2 in the post-test, a 
mean difference of 1.9 points. This indicates that children have increased their knowledge 
of the character words by 44%. 


Table 1 presents the percentage of children who answered each item correctly in the pre- 
and post-test. It shows that while there is a statistically significant increase in identifying the 
synonyms correctly for most words, some have seen a particular increase. For example, the 
words “vacant” and “gargantuan” show a particular increase of children knowing their 
meaning at post-test. 


However, the results also show that children were already quite familiar with certain words 
such as “sheepish” and “impeccable”, and the percentage of children who identified 
synonyms correctly for “cunning” and “bumbling” was not significantly different between 
pre- and post-test. 


Table 1: Synonyms correctly identified 


Word Pre-test Post-test 
Sheepish* 70.5% 88.5% 
Vacant* 25.4% 83.6% 
Impeccable* 75.4% 86.9% 
Cunning 66.4% 73.0% 
Bumbling 71.3% 68.0% 
Gargantuan* 34.4% 86.1% 
Scrawny* 58.2% 81.1% 
Assertive* 28.7% 50.0% 
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*statistically significant difference between pre- and post-test 


Word pairing task 

The word pairing task also showed improvement in children’s vocabulary. Their scores 
increased on average by 2.2 points, from 3.9 at pre-test to 6.1 at post-test, an increase of 
56%. 
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Table 2 shows the percentage of children pairing each word correctly in pre- and post-test. 
While all words have seen an increase in the number of children identifying the word pairs 
correctly, there is a particularly large increase with words such as “bloodshot” which has 
seen an 82% increase in the number of correct answers and “shrivelled” which has seen a 
72% increase in correct answers between pre- and post-test. 


Table 2: Word pairings correctly identified 


Word Pre-test Post-test 
Bloodshot* 45.9% 83.6% 
Bulky* 68.9% 91.8% 
Innovative* 42.6% 58.2% 
Brazen* 33.6% 64.8% 
Shrivelled* 52.5% 90.2% 
Insecure* 41.8% 70.5% 
Flawless* 60.7% 91.0% 
Daft* 46.7% 61.5% 


*statistically significant difference between pre- and post-test 


Writing task 

Analysis of children’s writing data showed that overall, children used more words from the 
Narrative Journey after taking part. On average, children used 3.9 words before and 5.3 
words after taking part (see Table 3). This increase is particularly interesting as the words 
were available to children in the writing task prompt both in the pre- and post-programme 
writing task. This might suggest that children learned to use the words accurately during the 
programme and hence incorporated more in their writing. 


However, children used fewer synonyms of these words after the programme activities (4.0 
vs. 1.7). This might suggest that children favoured words included in the Narrative Journey 
set over their synonyms and used them at the expense of other words. Word collocations 
were not used by children before or after taking part, which may be due to the nature of the 
task. 


Table 3: Average number of words used in children’s writing before and after the 


programme 
Before taking part | After taking part 

Narrative Journey words* 3.9 5.3 

Narrative Journey synonyms* 4.0 1.7 

Narrative Journey word collocations | 0 0.06 


*statistically significant difference between pre- and post-test 
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Changes in attitudes towards vocabulary learning 

After taking part in the programme, more children felt that pictures help them to remember 
new words better and that if they know a lot of words, they can improve their writing. As 
Figure 1 shows, before taking part, just over half the children felt that pictures help them to 
remember new words better, compared to over three in five after taking part. Similarly, 
73.8% of children agreed that knowing a lot of words will make them better writers before 
taking part compared to 80.4% after taking part. 


In contrast, while there also was an increase in the number of children at post-test who 
agreed that learning words is fun, this difference was not statistically significant. 
Interestingly, there was a drop over time in the number of children who agreed that 
knowing a lot of words can make them a better reader, but again, this difference was not 
statistically significant. The percentage of children who agreed they find it hard to learn new 
words was similar in pre- and post-test. 


Figure 1: Percentage of pupils who agree with attitudinal statements before and after the 
programme 
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These findings are particularly interesting as more children started to think that knowing a 
lot of words can improve their writing, but they did not feel the same about reading. It is 
possible that the writing tasks have helped more children to see the benefit of learning 
vocabulary. Similarly, it is interesting that while children did not necessarily see learning as 
more or less fun after using the Narrative Journey, more of them feel that pictures help 
them learn new words. This might indicate that the novel aspect of using pictures for 
learning vocabulary was perceived by children as useful and contributing to their learning. 
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Did any children in particular benefit? 

In order to assess whether there were any groups of children who benefited from the 
Narrative Journey in particular, the sample was divided into “lower” (n= 50) and “higher” 
(n= 57) -scoring children based on their overall score in the pre-test. 


The results show that children who scored lower in the pre-test increased their knowledge 
of vocabulary more than their higher-scoring peers. Lower-scoring children increased their 
overall score by 4.6 points (from 5.3 to 9.9) and higher-scoring children increased their 
overall score by 3.3 points (from 10.9 to 14.2). Interestingly, while lower-scoring children 
had a significantly greater increase in their word pairing score than their higher-scoring 
peers (2.7, from 2.3 to 5.0 vs. 1.6, from 5.4 to 7.0), there was no difference in their scores in 
the synonym task. This might indicate that the word pairings have been particularly 
beneficial for the lower-scoring children to start learning new words, but they might have 
not have completely understood the meaning of the words yet. 


Figure 2: The average score in vocabulary tasks for lower and higher-scoring pupils 
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Conclusion 

The findings of this pilot evaluation indicate that children who used Mrs Wordsmith’s 
Narrative Journey increased their knowledge of vocabulary. This is particularly true for 
children who started with a lower vocabulary score. 


In addition to increased vocabulary knowledge, more children feel that pictures help them 
learn new words and that knowing a lot of words makes them better writers. 


Next steps 

While the findings suggest that children have learned more words, the small sample size and 
the absence of a control group mean that it is not possible to attribute the improvement 
solely to the programme and determine whether the images in particular have made a 
difference in children’s vocabulary learning. In particular, future evaluation should focus on 
establishing whether children have actually acquired the meaning of the word from the 
pictures and can use it in a sentence, rather than simply matching the letters and the image. 


Future research could also focus on exploring whether the Narrative Journey is particularly 
beneficial for one group of children based on demographic factors such as gender and 
socioeconomic status. It would also be useful to test whether picture-based vocabulary 
learning can be used to improve poorer readers’ skills. 


Mrs Wordsmith Narrative Journey: Pilot evaluation © National Literacy Trust 2018 8 


